DOCUMENT RESUME 



ED 451 392 



CE 081 548 



TITLE 


Guidelines for Conducting the Follow-Up Survey. National 
Reporting System for Adult Education. 


INSTITUTION 


American Institutes for Research in the Behavioral Sciences 
Washington, DC. 


SPONS AGENCY 


Office of Vocational and Adult Education (ED) , Washington, 
DC. Div. of Adult Education and Literacy. 


PUB DATE 


2001-03-00 


NOTE 


52p . ,* For other National Reporting System for Adult 
Education documents, see CE 081 549-550. 


CONTRACT 


VN97 0 12 0 01 


AVAILABLE FROM 


For full text: http://www.air.org/nrs. 


PUB TYPE 


Legal/Legislative/Regulatory Materials (090) 


EDRS PRICE 


MF01/PC03 Plus Postage. 


DESCRIPTORS 


Adult Basic Education; Data Collection; Educational 
Assessment; Evaluation Methods; Federal Regulation; 



*Followup Studies; ^Guidelines; Longitudinal Studies; 
^National Surveys; ^Outcomes of Education; Postsecondary 
Education; ^Program Evaluation; ^Research Methodology; 
Sample Size; Sampling; State Surveys; Statewide Planning 
IDENTIFIERS ^National Reporting System for Adult Education; Workforce 

Investment Act 1998 Title II 



ABSTRACT 



The National Reporting System (NRS) established reporting 
requirements for the adult education program required by Title II of the 
Workforce Investment Act. The NRS defines the measures local programs must 
collect, the methodologies for collecting them, and state reporting 
requirements. This document presents the guidelines for conducting the NRS 
follow-up survey. Following an overview of the guidelines and the document in 
section 1, section 2 explains all aspects of conducting the survey, beginning 
with the outcome measures to collect through the survey and how to identify 
students who should be included. This section also explains the time periods 
in which to collect the measures, focusing on the employment measures 
(dependent on students' exit dates) . The section continues with the issue of 
sampling and specifies when programs should include all students in the 
survey and when they should draw a sample of students . Included in the 
section is a presentation on basic sampling concepts and terms, followed by 
the guidelines for sampling to meet the NRS survey requirements. Section 3 
presents the states' administrative responsibilities for the survey, 
reporting requirements and quality control procedures, along with suggestions 
for alternative methods for collecting the follow-up measures. Three 
appendixes include the following: (1) the rationale for, and statistical 

information on, the sampling requirements; (2) sample surveys and examples 
for drawing a random sample and training staff; and (3) procedures for 
computing the statewide average from individual program data for federal 
reporting. (KC) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 




Guidelines for Conducting 
the Follow-up Survey 



Division of Adult Education and Literacy 
Office of Vocational and Adult Education 
U.S. Department of Education 



March 2001 



EDUCATIONAL RESOURCES INFORMATION 
/VL. j CENTER (ERIC) 
lA L h J S d ° cu , menl has been reproduced as 
o“ngT ' he PSrSOn ° r Or 9 an ' 2at 'on 

□ Minor changes have been made lo 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy 



2 



BEST COPY AVAILABLE 



National Reporting System 



Conducted by: 

Pelavin Research Center 
American Institutes for Research 
Washington , D.C. 
Contract # VN97012001 

Project Staff 

Larry Condelli, Project Director 
Vince Padilla 
January Angeles 
Rob Leland 
Renee Sherman 
Mark Kutner 

John Tibbetts (Consultant) 
Allison Farrar 



For: 

U.S. Department of Education 
Office of Vocational and Adult Education 

Trish McNeil, Assistant Secretary for 
Vocational and Adult Education 

Ronald Pugsley, Director 
Division of Adult Education and Literacy 

Mike Dean, Program Specialist 
Division of Adult Education and Literacy 



2 



General Procedures for Conducting the Survey 



Table of Contents 



I. Guidelines for Conducting the Follow-up Survey 1 

Overview of NRS Survey Requirements 1 

Overview of This Document 2 

NRS Survey Requirements: Executive Summary 3 

II. General Procedures for Conducting the Survey 4 

Who and When to Survey 4 

Identifying and Surveying Learners: Employment Measures 5 

Identifying and Surveying Learners: Receipt of Secondary Credential and 

Postsecondary Placement 7 

Selecting Students for the Survey: Universe or Sample Survey 7 

Sampling Students: Basic Concepts 8 

Sampling Students: NRS Guidelines 1 1 

Procedure for Conducting the Follow-up Survey 13 

Informing Students of the Survey 13 

When to Conduct the Survey 13 

Identifying Survey Respondents 14 

Survey Questions 14 

III. State Administrative Responsibilities for Conducting the Survey 16 

Survey Administration 16 

Survey Reporting 17 

Computing the State Average 17 

Reporting Follow-up Outcome Data to ED 18 

Survey Verification 18 

Alternatives to Surveying 19 

Employment Measures 19 

Attainment of a Secondary Credential 20 

Entry into Postsecondary Education 20 

Appendix A: NRS Sampling Guidelines 

Appendix B: Sample Surveys and Sample Survey Procedures 

Appendix C: Computing the State Average from Local Samples 



List of Exhibits 



Exhibit 1: NRS Core Outcome Measures 1 

Exhibit 2: Student Population and Collection Time for Core Follow-up Outcome 
Measures 4 

Exhibit 3: Employment Measure Flow Chart for Adults Enrolled and Left by the End 
of the First Quarter 6 

Exhibit 4: Quarterly Periods for Collecting Entered and Retained Employment for 
Program Year 2000-2001 7 

Exhibit 5: Glossary of Sampling Terms for the NRS 10 

Exhibit 6: Summary of Procedures for Collecting the NRS Follow-up Survey 12 

Exhibit 7: Sample Local Program Survey Report Form 18 




5 



National Reporting System: Conducting the Follow-up Survey 



Guidelines for Conducting the Follow-up Survey 
for the National Reporting System 



I. Overview of NRS Survey Requirements 

The National Reporting System (NRS) established the reporting requirements for the 
adult education program required by Title II of the Workforce Investment Act, the legislation 
that authorizes the federal adult education program. The NRS defines the measures local 
programs must collect, die mediodologies for collecting them and state reporting 
requirements. While the NRS includes student descriptive measures (e.g., student age and 
edinicity) and participation measures (e.g., contact hours), it is the student outcome measures 
that have the greatest impact on the adult education delivery system. Exhibit 1 summarizes 
the NRS core outcome measures. 



Exhibit 1 

NRS Core Outcome Measures 



Topic 


Measures 


Definitions 


Educational Gains 


• Educational gains 


Improvements in 
educational functioning 
levels in reading, writing, 
speaking and listening and 
functional areas 


Follow-up Measures 


• Entered employment 


Learners who obtain a job 
by the first quarter after exit 
quarter 




• Retained employment 


Learners who entered 
employment who are 
employed in the third 
quarter after program exit 




• Receipt of secondary 
school diploma or GED 


Learners who obtain a 
GED, secondary school 
diploma or recognized 
equivalent 




• Placement in postsecondary 
education or training 


Learners who enroll in a 
postsecondary educational 
or occupational skills 
program building on prior 
services or training received 



Along with the descriptive and participation measures, programs collect the 
educational gain measure while students are enrolled in the program. However, programs 
must collect the NRS “follow-up” measures— entered and retained employment, receipt of 
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secondary credential and placement in postsecondary education or training — after the student 
has left the program Furthermore, the follow-up measures are student goal dependent — 
that is, they apply only to students who designate one of these outcomes as a main or 
secondary goal for attending. The employment measures apply only to students with a goal 
of obtaining a job and the receipt of postsecondary credential applies only to students who 
enter with a goal of obtaining this outcome, for example. Section 231(e) (2) of WIA also 
requires that all programs in every state report these measures. The challenge for local 
programs is to locate students after they have left the program to obtain the outcome measure. 

The NRS Implementation Guidelines describe two methodologies for collecting the 
follow-up measures: data matching and surveys. Data matching relies on using other 
databases, such as unemployment insurance wage records, to obtain the measures by 
matching student records with the information in the database. States use this procedure to 
meet reporting requirements for participants of job training programs (Title I of the 
Workforce Investment Act). Unfortunately, most states currently do not have the ability to 
use other data sources to collect all of the follow-up measures on adult education students. 

For some of the measures, extant data sources do not exist or the state lacks the infrastructure 
to perform the data matching. Consequently, most states will have to collect at least some of 
the follow-up measures through a survey of learners. 



Overview of This Document 

This dociunent presents the guidelines for conducting the NRS follow-up survey. 

The next section explains all aspects of conducting the survey, beginning with a discussion of 
the outcome measures to collect through the survey and how to identify students that should 
be included. This discussion also explains the time periods in which to collect the measures, 
focusing on the employment measures, which are dependent on students’ exit dates. 

This section continues with a discussion of the issue of sampling and outlines the 
conditions when programs should include all students in the survey and when programs 
should draw a sample of students. Included in this section is a presentation on basic 
sampling concepts and terms, followed by the guidelines for sampling to meet the NRS 
survey requirements. 

Section III presents the states’ administrative responsibilities for the survey, reporting 
requirements and quality control procedures, along with suggestions for alternative methods 
for collecting the follow-up measures. There are three appendices. Appendix A provides the 
rationale for, and statistical information on, the sampling requirements. Appendix B has 
sample surveys and examples of procedures for drawing a random sample, a procedure for 
conducting the survey and advice on training survey staff. Appendix C describes the 
procedure for computing the statewide average from individual program data for federal 
reporting. 



1 Programs may count as entered employment students who obtain a job while enrolled, although the 
outcome is reported when the student exits. For a full description of the NRS requirements see National 
Reporting System for Adult Education: Implementation Guidelines , March 2001 . Office of Vocational and 
Adult Education, U.S. Department of Education. 
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NRS Survey Requirements: Executive Summary 



To meet the reporting requirements for adult education programs in the Workforce 
Investment Act, states must collect from every local program four measures on students after 
they leave the adult education program: entered employment, retained employment, receipt 
of a secondary diploma or GED and entry into postsecondary education or training. These 
measures are collected only from students who designate these outcomes as their goals upon 
entry into the program. States must measure entered employment within the first quarter 
after students’ exit quarter from the program and retained employment in the third quarter 
after exit. Retained employment is measured only for students who obtained a job within the 
first exit quarter and for students who entered with a goal of improved or retained 
employment. The other outcomes can be collected at any time up to the reporting deadline 
(December 31), although quarterly collection is recommended. 

All local programs must report on the measures. Programs that have 300 or fewer 
students in any outcome area should include all of these students in the survey, but programs 
with 300 to 5,000 students in each outcome goal area may draw a sample of at least 300 
students. Programs with more than 5,000 students with any outcome goals must draw a 
sample of at least 1,000 students. In all cases, programs should get a 50% response rate or a 
minimum of 150 students in each group, if sampling is used (500 students per group for 
programs with over 5,000). A telephone survey is recommended. 

The state is responsible for survey administration, which includes ensuring that local 
programs have management information systems capable of identifying the students to be 
surveyed, determining which programs can sample students, whether optional measures 
should be collected and developing the survey instrument. States must also provided training 
and technical assistance to local staff, implement quality control procedures and aggregate 
local data for federal reporting. States are to report the statewide mean percentages, 
conference intervals and response rates for programs that sample students. 
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II. General Procedures for Conducting the Survey 

Conducting a follow-up survey is a complex activity that requires good organization 
and planning. This section describes the NRS procedures for planning and conducting the 
NRS follow-up survey, including identifying who is to be included in the survey, whether 
and how to sample students for the survey, when to conduct the survey and the methodology 
for conducting it. The administrative issues concerning the survey instrument itself, 
monitoring of the survey procedures, training of staff conducting the survey and reporting of 
findings are also addressed. 

Who and When to Survey 

WIA, section 231 (e)(2) requires that all local programs in every state report on the 
core outcome measures. However, only students who designate a main or secondary goal 
related to the follow-up measures need to be surveyed. In addition, only the outcome related 
to the goal needs to be measured. For example, only students unemployed at entry who 
designated a main or secondary goal of “obtain a job” need to be surveyed for the entered 
employment measure and only learners with the goal of obtaining a secondary credential or 
entering postsecondary education need to be surveyed for those measures. Exhibit 2 
summarizes the student population to include for each follow-up measure and when to collect 
it. 



Exhibit 2 

Student Population and Collection Time 
for Core Follow-up Outcome Measures 



Core Outcome Measures 


Student Population to Include 


Time Period to Collect 
Measures 


Entered employment 


Learners unemployed at entry 
with employment goal who exit 
during the program year. 


First quarter after exit 
quarter. 


Retained employment 


Learners unemployed at entry 
with employment goal who obtain 
a job during first quarter after 
exit; and learners employed at 
entry with a goal of retained or 
improved employment who exit 
during the program year. 


Third quarter after exit 
quarter, measured for 
students exiting the last 
two quarters of the 
program year. 


Placement in postsecondary 
education or training 


Learners with a goal of entering 
postsecondary education or other 
training. 


Any time prior to the 
reporting deadline 
(December 31). 


Receipt of secondary diploma 
or GED 


Learners with a goal of obtaining 
a secondary diploma or passing 
GED tests. 


Any time prior to the 
reporting deadline 
(December 31). 



*Exit quarter is the quarter when the learner completes instruction or has not received instruction for 90 days and has no 
instruction scheduled. A job obtained while the student is enrolled can be counted, but is reported in the first quarter after 
exit quarter. 
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Identifying learners for the retained employment measure is more complex than the 
other measures, as this measure applies to the learners who had a goal of obtaining 
employment goal and who then entered employment. The measure also applies to learners 
who had a goal of retained or improved employment. Measurement of both employment- 
related measures — entered and retained employment — is complicated further by the fact that 
these measures are time sensitive: the outcome must occur during a specific time period. The 
learner must enter employment by the end of the first quarter after the exit quarter (defined as 
the quarter when the learner completes instruction or has not received instruction for 90 days 
and has no instruction scheduled). A job obtained while the learner is still enrolled may be 
counted but is also reported in the first quarter after exit quarter. Retained employment is 
measured by determining whether the learner has a job in the third quarter after the exit 
quarter. 2 

Identifying and Surveying Learners: Employment Measures 

Exhibit 3 illustrates the identification of learners with employment-related goals for 
the survey using a hypothetical group of learners who enter a program on July 1, 2000. Of 
these learners, 120 have employment-related goals: 100 of them are unemployed and have a 
goal of obtaining employment and 20 additional learners have a goal of improve or retain 
employment. For this group of students, instruction ends by the end of September 2000, 
making this quarter in which their exit date falls — the first quarter of program year 2000 — the 
exit quarter. 

For the 100 learners with a goal of obtaining a job, whether they entered employment 
is measured during the first quarter after the exit quarter or the period October 1, 2000 to 
December 31, 2000. In this example, 65 of the 100 learners obtained a job. The remaining 
35 learners no longer need to be followed, even if they enter employment in subsequent 
quarters. For the 65 learners who entered employment and the 20 additional learners who 
entered with a goal of retain or improve employment, retained employment is measured in 
the third quarter after exit, or the period April 1, 2001 through June 30, 2001 (the last quarter 
of the program year). 

The time lag for the retained employment measure creates a substantial challenge to 
local programs to keep track of the shrinking cohort of students and to survey them at the 
correct time. A further difficulty created by the time lag is that the follow-up period for 
retained employment and entered employment goes beyond the program year. Programs 
must collect retained employment in the next program year on learners who exit in the 
second program quarter and must collect entered employment on learners who exit in the 
fourth program quarter. 

Exhibit 4 provides an example of when programs must measure retained and entered 
employment, using the 2000-2001 program year (July 1, 2000 through June 30, 2001) as an 
example. The reporting deadline for this program year is December 31, 2001. 



2 The time period for the employment measures is identical to the corresponding measures under Title I of 
WIA to allow for data matching using the unemployment insurance wage record data base. 
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Exhibit 4 

Quarterly Periods for Collecting Entered and Retained Employment 
For Program Year 2000-2001 



Exit Quarter 


Collect Entered 
Employment by the end of: 


Collect Retained 
Employment by the end of: 


First Quarter 
(July 1-September 30, 
2000) 


Second Quarter 


Fourth Quarter 


Second Quarter 
(October 1-December 31, 
2000) 


Third Quarter 


First Quarter, Next Program 
Year (July 1 - September 30, 
2001) 


Third Quarter 

(January 1-March 31, 2001) 


Fourth Quarter 


Not Reported 


Fourth Quarter 
(April 1-June 30, 2001) 


First Quarter, Next Program 
Year (July 1 - September 30, 
2001) 


Not Reported 



Note that retained employment is not reported for students exiting in the third or 
fourth quarters of the program year (i.e., the two quarters from January 1 to June 30). 
Programs should measure entered employment for these students in the appropriate quarter 
(first quarter after exit), but do not have to measure retained employment in the third quarter 
after exit (quarters ending December 31 and March 31 of the next calendar year). These 
quarters end beyond the reporting period for the program year (December 31), so retained 
employment cannot be determined in time to meet the deadline for the annual report for the 
program year. Consequently, these students need not be surveyed and are not counted in the 
computation of the percentage of students retaining employment in states’ performance 
standards. 

Identifying and Surveying Learners: Receipt of Secondary Credential 
and Postsecondary Placement 

Identifying the learners for the receipt of a secondary diploma or GED attainment and 
placement into postsecondary education measures is much simpler than the employment 
measures. Learners who designate these as goals for attending are surveyed on those 
measures. Likewise, deciding when to collect these measures is easier, since the NRS does 
not require measurement of these outcomes at any specific time. Programs can assess these 
outcomes at any time during the reporting period and should determine the most appropriate 
time for collecting them. For example, placement in postsecondary education could be 
collected shortly after the fall or spring semesters, corresponding to start times of the local 
community college. 

Selecting Students for the Survey: Universe or Sample Survey 

The WIA requires that states assess the performance of all local programs on the core 
outcome measures. States must obtain these measures on students from all of their adult 
education programs. For the larger programs, however, the NRS does allow states the option 
to decide whether to include all the learners in these programs in the survey — that is, the 
universe of learners — or a sample of learners from the larger programs. 
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ERIC 



Sampling a group of students can be much less expensive than a universe survey, but 
is problematic to the degree that the response from the students to the survey is expected to 
be low. Sampling creates a degree of uncertainty or error in the findings from the survey - 
known as sampling error - which becomes quite large, if response rate is low. Consequently, 
the lower the response rate, the more difficult it is to make an estimate of the true value of the 
outcome measure for all students. Since the response rate for adult education students is 
expected to be low, the use of the universe of learners in the survey rather than a sample is 
advantageous, since without sampling there is no sampling error (see Appendix A). 

However, since most states and adult education programs have minimal resources to conduct 
a survey, the number of students to include needs to be kept a to a minimum, making 
sampling attractive for large programs. 

With these considerations in mind, the NRS guidelines are for programs not to 
sample students, but to survey all students in each outcome goal area that has 300 or fewer 
students. That is, all students should be identified by goal area (entered employment, retain 
or improve employment, enter postsecondary education, obtain secondary credential) and if 
the total number of students for the year for any outcome is 300 or less, all students in that 
group should be surveyed to determine whether they have achieved the appropriate outcome. 3 
The minimum acceptable response rate for the survey is 50 percent. Programs may sample 
students for the survey for any outcome group that has more than 300 students exiting during 
the year. 

For example, if a program has 200 students with a goal of obtain employment; all of 
these students who exit are to be followed to determine whether they obtained employment in 
the first post-exit quarter. If that same program had over 300 students in other follow-up goal 
areas, the program has the option to survey all of these students or to draw a sample of them. 
The next section presents the guidelines for sampling. 

Sampling Students: Basic Concepts 

Only programs that have more than 300 students in any follow-up outcome area may 
conduct the survey on a sample of students for those measures. The sample should be drawn 
according to the guidelines described below. We first present some basic concepts and 
terminology of sampling before presenting the guidelines. 

The goal of a sample survey is to obtain accurate information about a large group of 
people, known as a population , using a smaller, representative group. Using statistical 
models, the findings from the sample can be estimated to the population with a known degree 
of precision. Consequently, sampling is less expensive and more efficient when a survey of 
an entire population is too large to survey. The key to a successful sample survey is to be 
able to generalize to die whole population with a small degree of sampling error. The factors 
affecting sampling error include; 

• Sample size — the number of people to include in the sample; 

• Response rate — the proportion of the sample that responds to the survey; 



3 For entered and retained employment, only students who exit are included. For the other follow-up 
measures, all students with the goal should be included in the follow-up for program year 2000. However, 
beginning July 1 , 2001 only students who exit should be included in the follow-up for all measures. 
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• Sample selection — the procedure for drawing the sample; and 

• Effect size — the size of the difference or finding you expect. 

Each of these factors was considered in establishing the NRS sampling guidelines. 

Sample Size. The number of people to include in a sample is of critical importance 
since the statistical model underlying the sample needs a minimum size to produce an 
estimate with a small degree of error. All other things being equal, the larger the sample the 
better, because the size of the error continually decreases as the size of the sample increases. 
However, there is a level of diminishing returns — once a large enough sample size is 
achieved the error decreases little and larger samples are not usually worth the additional 
cost. While the specific minimum or maximize number of responses needed for a survey 
depends on several factors, the minimum number might typically be around 100 and the 
maximum number around 500. However, the number to include in the initial sample depends 
on the expected response rate. 

Response Rate. Nonresponse to a sample survey increases the uncertainty of in die 
results in two ways: the overall number of students responding is lowered, increasing the 
error and we do not know how nonrespondents would have scored on the outcome measures. 
It is unlikely that die nonrespondents to the survey have the same characteristics as die 
respondents. For example, an employment rate would be badly overestimated with high 
nonresponse, if most of the respondents are employed and most of the nonrespondents are 
unemployed. Some types of sample selection make response rates even more critical. 

Sample Selection. The simple and most common way to select a sample is to draw a 
simple random sample. In a random sample, every person has an equal probability of 
selection. Random sampling is usually preferred since it is the easiest type of sampling to 
perform, helps keep the sampling error low and produces results that are easy to generalize to 
the population as a whole. With simple random sampling, the survey findings directly reflect 
the population. 

Another type of sample is a stratified sample, where people are grouped along a 
dimension believed to be important to the survey. Different groups of people are first 
identified and then sampled separately. Ethnicity, race and sex are common ways samples are 
stratified. This type of sample is used to ensure that the stratified groups are adequately 
represented in the sample and in some cases can reduce the sampling error. The disadvantage 
of this procedure is diat it may require that the sample be weighted to reflect die different 
representation of the stratified group in the sample. 

Effect Size. Effect size refers to how large an effect is expected to be found in 
the survey population. For example in designing the NRS sample, knowing what 
percentage of students with an employment goal can be expected to get a job would help 
decisions about sample size. A larger effect creates the need for a larger sample to keep 
the standard error low. The difficulty in the NRS is that there are little data available that 
will guide us in estimating the size of the effects (e.g., number of jobs, GEDs, 
postsecondary enrollment) that will be found. 

Exhibit 5 is a glossary of sampling terms that is helpful in planning the survey 

sample. 
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Exhibit 5 

Glossary of Sampling Terms for the NRS 



Population — The larger group to which the survey results are to be inferred 
and which a sample represents. For example, a population can be all adult 
education students, all students who had a goal of employment who attended 
an adult education program or all residents of the U.S. 

Universe — All members of a group, such as all students who students 
enrolled in an adult education program who want to pass the GED tests. 

Sample — A group drawn systematically from the population or universe of 
interest. 

Random sampling — A sample drawn using a method whereby every 
member has an equal probability of being selected. 

Stratified sample — A sample that is grouped according to a dimension 
believed to be important, such as race, sex or age. The sample is then drawn 
separately for each group. 

Achieved or realized sample size — The number of entities or people who 
actually provided information to a survey or study. 

Response rate — Proportion of the sample that actually responded to the 
survey or study, usually reported as a percent (total responded divided by total 
in the sample). 

Standard error — A measure of the degree of uncertainty for inferring to a 
population from a sample. This number is used to compute the confidence 
interval. 

Effect size — The size of the difference or effect being studied, such as the 
percentage achieving a goal or a mean number. 

Confidence interval — a numeric interval in which the true number obtained 
for a population from a sample falls. The interval is stated with a degree of 
probability, typically 95 percent. For example, if a survey using a sample of 
300 found a 25 percent effect, the confidence interval is 20 percent - 30 
percent (25 plus or minus 5), meaning that there is a 95 percent certainty that 
the true percentage in the overall population falls within that inclusive range. 
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Sampling Students: NRS Guidelines 

In developing the sampling guidelines for the NRS, we considered not only the 
statistical considerations outlined above, but also the minimal resources likely to be available 
to local adult education programs to conduct the survey, as well as the expected low response 
rate from the adult education student population. Our goal was to make the sampling for the 
survey as simple and as inexpensive as possible under these conditions, but yet still achieve a 
sampling error low enough for the findings to be meaningful. With these considerations in 
mind the following guidelines are the minimum requirements for conducting the NRS follow- 
up survey for programs that have over 300 and fewer than 5, 000 students in a program year 
within one or more of the outcome groups 4 

1. The sample must include a minimum of 300 students for each of the outcome 
areas; 

2. A simple random sample should be drawn; 

3. The response rate must be at least 50 percent. If it proves to be impossible to 
achieve a 50 percent response rate, the program should continue surveying so that 
there are at least 150 students in each outcome group. 

Programs that have more than 5, 000 students in a program year in any outcome group 
should sample at least 1,000 students with a 50 percent response rate or achieve a realized 
sample size of at least 500 students for each outcome group. 

These sampling requirements will produce a maximum standard error of four and 
offer no adjustment for nonresponse. States may use other methods for sampling to meet 
NRS requirements upon approval from the U.S. Department of Education, but alternative 
procedures must meet the minimal criteria for statistical precision presented here. For 
example, the maximum standard error is to be four and a minimum 50 percent response rate 
is required. Appendix A provides more information on the statistical rationale for these 
requirements. Exhibit 6 summarizes the procedures for the NRS follow-up survey. 

Measuring Retained Employment in a Sample. The retained employment measure 
needs to be collected from (1) students who had an goal of obtain employment and who got a 
job within the first quarter after exit and (2) students who had a goal of improve or retain 
employment. Thus, the group on which to collect this measure should include all survey 
respondents who report obtaining a job in the first quarter and all students with the improve 
or retain employment goal. 

Measuring the NRS Optional Measures. The NRS includes a set of secondary, 
optional measures that states may report and collect on their students. These measures 
are in the areas of community and family literacy and include an employment measure 
for students receiving welfare. While some of the optional measures can be collected 
while the student is enrolled, others should be collected through a follow-up survey. For 



4 Remember that the number of students with each follow-up outcome goal determines whether as sample 
may be drawn, not the total number of students served by the program. As discussed earlier, if the program 

has 300 or fewer students in any outcome area, all of the students (the universe) are to be surveyed. 
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Exhibit 6 

Summary of Procedures for Conducting the NRS Follow-up Survey 



1. Identifying the Survey Population 

• In each program in the state, identify (1) all unemployed students with a goal of obtaining a job who 
exited, (2) employed students with a goal of keeping or improving their current job who exited, (3) 
students with a goal of obtaining a secondary diploma or passing the GED and (4) students with a goal of 
entering postsecondary education or training. Beginning the program year that starts July 1 , 2001, 
include only students who exited during the program year. 

• Obtain the exit date or quarter for students with employment goals to identify follow-up date. 

• Exclude from the retained employment measure all students with an employment goal who exited in the 
third or fourth quarters. 

• If the state is requiring collection of the NRS optional measures, the students with goals related to these 
outcomes should also be identified. 

2. Selecting Students for the Survey 

• Survey all students in any outcome group that has 300 or fewer students in the program year. 

• A simple random sample of students may be surveyed for any outcome group that has more than 300 
students. 

• Programs should draw a minimum sample size of 300 for each group that has 301 -5,000 students. 

• Programs should draw a minimum sample size of 1,000 for each group that has over 5,000 students. 

• A minimum response rate of 50 percent is required. If less than 50 percent response rate is achieved, the 
program must survey for any group that was sampled until it has responses from at least 150 students (or 
500 students for the largest programs). 

3. Collecting the Data 

• Quarterly data collection is recommended for all measures, since entered employment must be recorded 
for the first quarter after program exit and retained employment in the third quarter after exit. 

• Retained employment is to be collected only for students exiting in the first two quarters of the program 
year. 

4. Reporting the Findings 

• Each program must report for each outcome group: the total number of students with the goal; the total 
number of students responding to the survey; the number and percentage of students achieving each 
outcome; and the total number of students included in the survey or if sampling was employed, the total 
number sampled. 

• The state is to compute a weighted state average of students achieving each outcome by aggregating 
from the individual programs: the total number of students in each outcome group, response rates and the 
number and percentage achieving each outcome. The state must also report the state confidence 
interval if any local programs use sampling. 
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example, family literacy programs could collect measures of improved family literacy 
activities while the adult learner is enrolled in the program, or could conduct a follow-up 
survey to ask about improvements. Measures such as employment of students who were on 
welfare and registering to vote, however, would usually have to be collected in a follow-up 
survey. 



If the state elects to have local programs use the follow-up survey to collect the 
optional measures, it is recommended that programs follow the same procedures as for the 
survey on the core measures. That is, the program should identify the population of students 
to whom the measures apply through student goals (e.g., register to vote, read more to 
children, get off of welfare). Where there are 300 or fewer students in a goal area in a 
program year, the program must include all of the students in the survey. Where there are 
more dian 300 students in a goal area in a program year, the program may sample, according 
to the guidelines outlined above for the core measures. 

Procedure for Conducting the Follow-up Survey 

Surveys are conducted by mail, telephone or through an in-person interview. The 
latter option is extremely costly and is not feasible for adult education programs. A mail 
survey, while inexpensive and very easy to administer, normally produces an unacceptably 
low response rate — typically around 10 percent. Consequently it is highly recommended that 
local programs conduct the follow-up survey by telephone. The telephone survey may be 
supplemented with a mail survey and by information collected while the student is enrolled. 
For example, programs could supply students with an employment goal with a postcard to 
return when they obtain a job, or the teacher could obtain this information while the student 
is still enrolled. Programs may conduct only a mail survey, at the state’s option, if a 50 
percent response rate can be achieved. 

Informing Students of the Survey 

It is very important to the success of the survey that students know they may be 
contacted later and asked about their outcomes, since the main challenge in conducting a 
survey is finding the students after they leave the program. As discussed above, as many 
students as possible must be surveyed, since the response rate is critical to the validity of the 
survey findings. For this reason, programs should inform students with follow-up goals 
about the survey upon entry into the program and collect extensive contact information about 
them, such as addresses and phone numbers of relatives or others who may know of the 
students’ whereabouts over time. In addition, students should be encouraged to provide new 
addresses and phone numbers when they move. Programs should implement procedures to 
update this information periodically while the student remains enrolled. These procedures 
will greatly assist locating students months later when the survey is conducted. 

When to Conduct the Survey 

The entered and retained employment measures are tied to calendar quarters. To 
collect diese measures programs must identify an exit quarter and then collect the measure in 
the first or third post-exit quarter. For this reason, the simplest time to conduct the survey is 
quarterly. If quarterly collection is conducted, the survey should begin no sooner than the 
last mondi of the quarter and be completed within three months (one quarter). The 
attainment of postsecondary credential and entering postsecondary education measures are 
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not time bound, so while they can be collected at any time during the reporting period, the 
easiest option is to also collect them by quarter. The program should determine the optimal 
time to collect these measures. For example, it may be advisable to collect the entry into 
postsecondary measure in the fall quarter when most students enter community college. If 
there are scheduled time when the GED tests are given, the program could measure that 
outcome for students with the secondary credential goal around that time. It is recommended 
that equal numbers of students be surveyed each quarter. For example, if 300 students are to 
be surveyed, about 75 students should be surveyed each quarter. 

While quarterly data collection is strongly recommended, states may set other times 
to conduct the survey, if they are more convenient or cost-efficient. For example, the 
program could conduct continuous, ongoing surveys or could conduct the survey biannually 
(such as in December and June). However, the time lag to contact students after they exit the 
program should be as short as possible. The longer the time, the lower the response rate 
(since some students will move) and the less valid the data are likely to be. 

Identifying Survey Respondents 

Each program’s management information system needs to have the capability to 
produce a list of students eligible for the survey. The lists should be organized by students’ 
outcome goal and include each student’s identifying and contact information. The list should 
include exit dates or quarters for students with employment goals. As explained above, if 
there are 300 or fewer students in any outcome area, the program should survey all of these 
students. If there are over 300 students in any area, a sample may be drawn. A simple 
random procedure should be used to draw the sample. If possible, an approximately equal 
number of students should be drawn for each survey collection time (e.g., a quarter of the 
total student sample each quarter). Appendix B illustrates a procedure for drawing a random 
sample for the NRS survey. 

Survey Questions 

The survey itself should be brief, since people are unlikely to want to spend more 
than a few minutes responding to it. After explaining the purpose of the survey, the 
interviewer need only ask the specific questions related to the student’s outcome to collect 
the NRS measure. Below are sample questions for each measure. 

• Entered Employment — Have you gotten a job since you left the program? If 
yes, when? Where do you work? 

• Retained employment — Are you currently employed? If yes, where do you 
work? 

• Enter postsecondary education or training — Have you enrolled in a 
community college, a four year college, another postsecondary school, a job- 
training course or another education or training course since you left the 
program? If yes, where have you enrolled? 

• Obtained a postsecondary credential or passed the GED — Have you passed 
the GED tests or obtained a high school diploma since you left the program? 
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Once students are contacted, states or programs may wish to use the opportunity to 
collect additional information using the survey, such as the student’s satisfaction with the 
program or other outcome measures. Appendix B includes two surveys, one that collects 
only basic information and another that also collects NRS optional measures and other 
information in which programs may be interested. 



General Procedures for Conducting the Survey 



15 



National Reporting System: Conducting the Follow-up Survey 



III. State Administrative Responsibilities for 
Conducting the Survey 

To help control the quality of data collected, the NRS requires that the state direct the 
management and administration of the NRS follow-up survey. State control of the survey 
process will ensure that local programs identify and select the appropriate students for the 
survey, ask comparable survey questions, follow the appropriate methodology and 
consistently report findings. States also must implement procedures to verify the validity of 
survey findings. This section presents the NRS guidelines in each of these areas. 

Survey Administration 

State staff may directly conduct the survey, hire a third party contractor or have local 
program staff conduct the survey, as long as data are collected from all funded programs . 5 
However, regardless of who actually collects the survey data, the state is to establish and 
direct the general survey procedures to ensure conformance with the NRS guidelines. The 
specific state responsibilities are to: 

• Ensure the state or local management information system can generate student 
lists with contact information by outcome area. For students with an employment 
goal, the lists should also include the exit date. 

• Determine whether any or all of the NRS optional measures should be collected 
through the survey. 

• Determine which programs, if any, can sample students for the survey. 

• Determine whether the survey is to be conducted by telephone or mail. 

• For programs that sample, ensure that NRS sampling procedures are followed, 
including that the program draws a simple random sample, draws the minimum 
sample size and achieves the minimum response rate. 

• Establish the time when the survey is to be conducted, such as monthly, 
biannual ly or quarterly. 

• Provide the actual survey instrument or questions programs should ask. Local 
programs should not develop their own survey, but the state should provide a 
standard statewide survey. See Appendix B for sample surveys. 

• Ensure programs comply with NRS reporting requirements for the survey. 

• Conduct studies to verify the validity of survey data (see below for more 
information). 



5 Section 231 (e)(2) of Title II ofWIA requires states to assess the performance of every funded local 
program using the core indicators. 
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To meet these responsibilities, states need to implement procedures to train local 
survey staff on survey requirements, provide ongoing technical assistance and monitor 
compliance with survey procedures. 

Survey Reporting 

The state must report to the U S. Department of Education (ED) the overall state 
percentage of students who achieved each of the follow-up outcomes. As required under 
WIA, ED will report these measures to the Congress as indicators of the overall performance 
of the adult education program. The measures are also one factor that ED will use to 
determine the state’s eligibility for incentive awards available under WIA. 

To compute the state overall measures for each outcome, the state will have to 
aggregate each of die measures from every local program to compute a state average. Each 
local program must report the following information to the state to enable computation of the 
state average. 

• Total number of students in each outcome group served during the year (for 
employment measures, die number who had an employment goal and exited; for 
other follow-up measures, count all students until the program year beginning 
July 1, 2001. Thereafter, count only exited students for all follow-up measures); 

• If die program sampled, the total number of students sampled; 

• Number of students who responded to the survey (the realized or actual sample 
size) and the response rate; and 

• The percentage of students achieving each outcome. 

Using the individual program percentages, the state must compute and report die state 
average on NRS Table 5 (see Chapter V, Measures and Methods for the NRS: 
Implementation Guidelines). Exhibit 7 shows a sample reporting form for local programs to 
report survey results. 

Computing the State Average 

Using the information from every local program, the state is to compute a state 
average. Appendix C explains how to calculate the average and includes the statistical 
formula. The statistical procedure produces a weighted average and gives more weight to 
data from large programs. That is, data from larger programs will have a greater impact on 
the state average than data from smaller programs, since the larger programs serve more 
students and the sample from these programs represent a greater number of students. The 
weighted average also allows adjustment for nonresponse, since NRS policy allows the 
assumption that nonrespondents achieve die outcome in the same proportion as respondents 
(see Appendix A). If any local programs use sampling, the state must also compute and 
report the range of the estimate, or the confidence inten’al , for the state average. For 
example, a state must report that 25 percent, plus or minus six percent, achieved the outcome. 
The overall response rate for the state should also be computed for each outcome measure by 
summing the total number of students who responded and dividing it by the total number of 
students included in die survey. 
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Exhibit 7 

Sample Local Program Survey Report Form 



Outcome 

Measure 


Total Eligible 
Students with 
Main or 

Secondary Goal 


Total Number of 
Students 
Sampled 


.Total Number of 
Students 
Responding 


Number 

Achieving 

Outcome 


Response Rate 
(Total 

Responding/ 
Total Sampled) 


Entered 

Employment 












Retained 

Employment 












Receipt of 
Secondary 
Credential 












Entered 
Postsecondary 
Education or 
Training 













Reporting Follow-up Outcome Data to ED 

States are to use NRS Table 5 6 to report the follow-up outcome data, including the 
total number of participants in each outcome group, the number that achieved the outcome 
and the percentage achieving each outcome. To assist ED in judging the quality and validity 
of the follow-up data conducted through the survey, states need to provide the information 
listed below about how the survey was conducted. 

• The total number of programs that sampled students. 

• The total size of the sample drawn and total number of students contacted. 

• The state confidence interval for each measure where sampling was used. 

• The state response rate for each measure. 

• The time period when the survey was conducted. 

• The methodology used to conduct the survey. 

Survey Verification 

Surveys such as the one proposed for the NRS, rely on students to report on their 
achievements. In die social sciences, diis type of self-report data is often believed to be of 
questionable validity, since respondents may feel pressured to report a successful outcome. 
To gauge the extent of this potential bias and to improve the quality of data, validity studies 
are often conducted to verify survey findings. 



6 See National Reporting System for Adult Education: Implementation Guidelines. March 2001. Office of 
Vocational and Adult Education, U.S. Department of Education. 
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As a means of data quality control, states must verify at least one of the surveyed 
follow-up outcomes annually. This verification should be done through a random sample of 
students in the outcome area. The following sources may be used as a means of verifying the 
data and states are encouraged to develop additional ways to verify survey data. 

• Employment — Contact employers or use data from the unemployment insurance 
database. 

• Secondary Credential Attainment — Obtain data from the state GED database. 

• Entry into Postsecondary Education or Training — Contact local community 
colleges and training programs to verify enrollment. 

To validate data, the state should select the outcome area to validate and obtain lists 
of students who have achieved the outcome from each local program. Using these lists, the 
state should draw a statewide random sample to achieve a realized sample size of 500. The 
response rate should be no less than 50 percent. State staff should contact the verification 
sources and report the: 

• Outcome area and total number of students sampled for verification; 

• Response rate; 

• Method used to select the sample; 

• Number of students for which data were validated; 

• Number of validated as true; and 

• Sources used to validate the data. 

States need only validate one outcome annually and validate a different outcome each year. 

Alternatives to Surveying 

Although states are not required to use a survey to collect the NRS follow-up 
measures, in many cases this methodology may be the best or the only alternative. As is 
apparent from this monograph, however, surveying is a complex and time-consuming 
activity. In particular, the burden on local programs can be substantial, as many programs 
may not have sufficient resources to conduct a survey. Consequently data quality may suffer. 
This section provides some suggestions for other ways of collecting these data. States are 
encouraged to consider and develop these and other ways to collect the NRS follow-up 
measures. 

Employment Measures 

Employment training programs funded under Title I of WIA use the wage records 
from each state’s unemployment insurance database to report participants’ employment 
outcomes. The states can also use this database to determine the entered and retained 
employment of adult literacy students to meet NRS requirements. Use of this procedure 
requires the state to create a statewide database with the records of exited adult education 
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students who had employment goals and then match these records quarterly with records on 
the UI database. A match indicates the student is employed. This data matching procedure 
requires a common identifier, usually students’ Social Security numbers . 7 Unfortunately, 
most states do not currently have the infrastructure to support this option for adult education 
students. 

Another approach to obtaining employment information is to have direct contacts 
with local employers. If the program works with a finite number of employers who are adult 
education students or if the program coordinates with job developers in an employment 
program, these sources can provide information on which students are employed and when. 
However, only small programs in small communities or programs that have few students with 
employment goals are likely to be able to use this approach. 

Another way to collect employment data, at least for some students, is to have an 
agreement with local one-stop centers to identify co-enrolled students. The one stop 
center can inform the local program of which students are co-enrolled. Since one stop 
centers must determine employment of their participants using a data matching 
methodology, they can notify the local program of the job attainment of these students. 
The adult education program can then record this information in its data base. 

Attainment of a Secondary Credential 

Rather than survey students, the most direct way to collect this measure is to match 
students records with the state GED test data base and other state data sources that record 
awarding of secondary diplomas (such as adult high schools). On a periodic basis (e.g., 
annually, quarterly) the state can create a database of students with a goal of attainment of 
secondary credential and then match this database with the statewide GED or equivalent 
databases. A match would indicate attainment of the credential, which the state can count for 
NRS reporting. The state then would not need to survey these students. 

Another way to obtain GED test results is to make an arrangement with the state’s 
GED testing center to have the names of students who pass the tests submitted to the local 
adult education program or to a central state office. These names can then be matched to 
students in the local data base and counted for NRS reporting. 

Yet a third way to obtain GED test results is to have the students sign a release to 
have test scores sent directly to the local adult education program. The program can then 
record test completion in its data base. 

Entry into Postsecondary Education 

In many communities, adult education students advance into local community 
colleges, four year colleges or other training programs that are known to program staff. 

Instead of surveying students with a goal of entry into postsecondary education, programs 
could contact these colleges and training programs directly to determine whether their adult 
education students enroll. The program can designate a staff person, such as a transition 



7 For more information on data matching sees Chapter IV, National Reporting System for Adult Education: 
Implementation Guidelines. June 2000, and Report on the Pilot Test for the National Reporting System for 
Adult Education. 1999. Office of Vocational and Adult Education, U S. Department of Education. 
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coordinator, to maintain contacts with local colleges and provide the names of students they 
expect to enroll. The transition coordinator can then contact the college or school 
periodically to obtain updated enrollment lists and determine who has enrolled for NRS 
reporting. While this approach is not feasible in large urban areas, it could be used in small 
and medium-sized communities. 
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Appendix A 

The NRS Sampling Guidelines 

The goal of a survey is to make an estimate of how a large group of people will score on 
measures by asking only a sample of people drawn from this population. Statistical theory allows us 
to make this estimate with a degree of uncertainty or sampling error. The main contributors to error 
in a simple random sample, such as will be drawn for the NRS, are determined primarily by (1) the 
size of the sample and (2) the degree of nonresponse to the survey. Each of these factors is discussed 
below in the context of the NRS sampling guidelines. 

Sample Size 

A simple fact of sampling is the larger the sample the smaller the error. However, there is a 
level of diminishing returns — once a large enough sample size is achieved the error decreases little. 
To non-statisticians this number seems surprising small: the error decreases at a markedly lower rate 
once the sample size reaches around 500. 

The formula for computing the standard error for a percentage at the 95 percent confidence 
level, such as would be computed in reporting the NRS outcomes, is: 



i n wo- p) 

where p = the proportion found and n = the number of students in a sample. 

Exhibit A-l presents the standard errors for selected achieved sample sizes and percentages. For 
example, if a survey found that 25 percent of students from a sample of 300 students obtained a job, 
the standard error would be 2.50. 8 

Survey results are reported with a degree of certainty, known as a confidence interval, which 
by convention is an interval where there is a 95 percent certainty within which the number of interest 
falls. The confidence interval is computed by doubling the standard error and adding and subtracting 
the product form the mean. For the example above, where a sample of 300 found a 25 percent effect, 
the confidence interval is 20 percent - 30 percent (25 plus or minus 5). This interval is interpreted as 
meaning that there is a 95 percent certainty that the true percentage in the overall student population 
is between 20 percent and 30 percent, inclusive. 



8 Note that when there is no sampling, the standard error is zero. Thus, programs that do not sample students for the 
NRS have no standard error or confidence interval to report. There is no error because students responding to a 
universe survey are not drawn from a larger group of students. 
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Exhibit A-1 

Standard Errors by Achieved Sample Size for 
Percentages of Students Achieving Outcomes 



Percentage of Students Achieving Outcome 



Achieved 
Sample Size 


5 


10 


25 


50 


75 


90 


95 


100 


2.18 


3.00 


4.33 


5.00 


4.33 


3.00 


2.18 


150 


1.78 


2.45 


3.54 


4.08 


3.54 


2.45 


1.78 


200 


1.54 


2.12 


3.06 


3.54 


3.06 


2.12 


1.54 


250 


1.38 


1.90 


2.74 


3.16 


2.74 


1.90 


1.38 


300 


1.26 


1.73 


2.50 


2.89 


2.50 


1.73 


1.26 


350 


1.16 


1.60 


2.31 


2.67 


2.31 


1.60 


1.16 


400 


1.09 


1.50 


2.17 


2.50 


2.17 


1.50 


1.09 


450 


1.03 


1.41 


2.04 


2.36 


2.04 


1.41 


1.03 


500 


0.97 


1.34 


1.94 


2.24 


1.94 


1.34 


0.97 


750 


0.80 


1.10 


1.58 


1.83 


1.58 


1.10 


0.80 


1000 


0.69 


0.95 


1.37 


1.58 


1.37 


0.95 


0.69 



Note: The table shows standard errors for the 95 percent confidence level. The standard error of a percentage larger than 50% 
is the same as the standard error of 100 minus that percentage (so the standard error of 75% is the same as that of 25%). 

Setting the NRS Minimum Sample Size 

The other factor that affects the setting of sample size is the cost of the survey. Since larger 
samples are more costly, the key to establishing an acceptable minimum sample size is to find the 
sample size one can afford, while keeping the error as low as possible. Within the NRS, it is 
recognized that states and programs have minimal resources to conduct surveys, yet the sample size 
must be sufficiently large so that the confidence interval is not too wide. Another factor affecting the 
decision of sample size is that we do not know the size of the percentages we will obtain for the 
outcome measures — i.e., what percentage of students will obtain jobs, enter postsecondary 
education and attain a postsecondary credential. 

With these considerations in mind, we used the following criteria to set the minimum sample 
size for the NRS: 

1. The largest standard error we would tolerate is about four (plus or minus 8 confidence 
interval). 

2. The sample size we would ask local programs to use would be a small as possible to keep the 
cost low. 

3. As do not know what percentage of students would achieve each outcome, we made the most 
conservative estimate. Therefore we assumed that 50 percent would achieve each outcome, 
which produces the highest standard error (see Exhibit A-1). 

4. The minimal acceptable response rate was 50 percent, again to keep costs low. 

Using these criteria, a realized sample size of 150 students is needed for the NRS (see Exhibit A-1), 
with an initial sample size of 300 drawn (assuming half will not respond). 
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With this minimum sample size, if 50 percent of students are found to enter employment, for 
example, the confidence interval is between about 42 (41.84) percent and 58 (58. 16) percent. This 
percentage produces die highest error that could be obtained with this sample size. If 75 percent of 
students sampled obtained a job, for example, the standard error is 3.54. Since percentages other 
than around 50 percent produce smaller standard errors, states can use a different sample size, upon 
approval from die Department of Education, as long as the realized sample can produce a standard 
error of four or less. States must also achieve a minimum 50 percent response rate. 

Response Rates 

The other factor that affects sampling error is the response rate. This error is due simply to 
the fact that we do not know how people who did not respond to the survey would have performed on 
the measures (e.g., gotten a job, entered postsecondary education). For example, the nonrespondents 
to the NRS employment survey may have a substantially lower or higher rate of employment than do 
the respondents. By not taking this possibility into account, the estimate of employment rate, based 
only on the respondents, might be vastly overstated or understated. 

Exhibit A-2 shows die 95 percent confidence intervals on some percentage of interest (like 
unemployment rate) for 50 percent nonresponse. In this table, we have made two different 
assumptions about the nonrespondents. For the lower bound of the confidence intervals (P low), we 
have assumed diat none of die nonrespondents had the characteristic of interest (e.g., 0 percent 
obtained a job). In the upper bound of the confidence intervals (P high), we have assumed that all of 
the nonrespondents had the characteristic of interest (e.g., 100 percent obtained a job). For example, 
assume a program drew an initial sample of 300 and reached 150 students. If 25 percent of diese 
students obtained a job, the true confidence interval taking into account all nonrespondents, is 
between about 9 percent and 66 percent (8.96 - 66.04). 9 

Exhibit A-2 

Confidence Intervals for Varying Respondent Sizes and Percentages, 

50 Percent Response Rate and Nonresponse Assumptions 





Response Rate = 50% 


Nonresponse Rate = 50% 






Percentage Achieving Outcome 


Number of Students 
Selected for the Survey 


5.00 


10.00 


25.00 


50.00 


Desired 


Achieved 


P Low 


P High 


P Low 


P High 


PLow 


P High 


P Low 


P high 


100 


50 


0.00 


55.58 


0.76 


59.24 


6.38 


68.62 


17.93 


82.07 


200 


100 


0.32 


54.68 


2.00 


58.00 


8.17 


66.83 


20.00 


80.00 


300 


150 


0.72 


54.28 


2.55 


57.45 


8.96 


66.04 


20.92 


79.08 


400 


200 


0.96 


54.04 


2.88 


57.12 


9.44 


65.56 


21.46 


78.54 


500 


250 


1.12 


53.88 


3.10 


56.90 


9.76 


65.24 


21.84 


78.16 



Note: P low assumes 0%of nonrespondents achieved the outcome, P High assumes 100% of nonrespondents achieved the 
outcome. 



9 Note that unlike standard error, error of the estimate due to nonresponse applies whether or not students are 
sampled. The uncertainty from nonresponse stems from the lack of knowledge of nonrespondents - and both 
universe and sample surveys may have some degree of nonresponse. 
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It is this effect on the precision of the results that makes nonresponse so problematic. For 
this reason, follow-ups of the nonrespondents are essential to try to reduce the level of nonresponse 
as much as possible. Bearing in mind the substantial cost to do follow-ups, however, we decided that 
a 50 percent response rate was the minimum acceptable for the NRS. 

Adjusting for Nonrespondents 

For the NRS, as widi other surveys, a decision had to be made on how to adjust for 
nonresponse. There are four basic ways to adjust for nonresponse. 

1. Assume none of the respondents achieved the outcome (P Low in Exhibit A-2). This assumption 
is the most conservative way to adjust for non-response, as it lowers the estimate substantially 
when nonresponse is high. 

2. Assume all of die respondents achieved the outcome (P High in Exhibit A-2). This assumption 
produces the least conservative way to adjust for non-response, as it raises the estimate when 
nonresponse is high. 

3. Assume nonrespondents are exactly like respondents. This assumption means diat the effect 
found among respondents would be the same as among nonrespondents. For example, if 25 
percent of respondents report having gotten a job, we assume 25 percent of nonrespondents also 
got a job. Although this assumption is probably not true (i.e., that respondents and 
nonrespondents are exactly the same), it is how nonresponse is handled in many surveys and 
without evidence to the contrary, is a defensible, middle-of-the-road approach. 

4. Make a statistical estimate of how the nonrespondents would score based on other data. This 
procedure produces the most accurate estimate, if there exists data that are predictive of the 
outcome. Unfortunately, we know of no data that programs could readily use to make these 
adjustments. In addition, the cost of doing such adjustments is prohibitive within the NRS. 

For the NRS, we decided to adjust using the third option above. We assume nonrespondents 
and respondents would score alike on the outcomes. While this assumption may not be accurate, in 
the absence of other data it is acceptable practice for the purposes to which the NRS data will be 
used. 
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National Reporting System: Conducting the Follow-up Survey 



Sample Local Follow-up Survey for Core Measures 





Enroll?^ent 


iiliiiisiiiiiii 


lili 


■. 









Hello. My name is . I work for . We’re calling people who have recently attended classes at 

our adult education program to find out what happens t them after they leave us. We want to know how you liked the classes 
you took and how adult education classes have affected you, your family, and your job. 

It should take no longer than 10 minutes to answer my questions. Do you have time now for me to ask these questions? 
(Reassure the respondent that any information given to us will be strictly confidential). 

First, I’d like to make sure I have the correct information about the class you took. 

A-1. I understand that you were in (TEACHER’S NAME)’s class at (LOCATION. Is that correct? 

□ Yes 

□ No [ Obtain correct inform ation] 

A-2. Did you attend teacher’s class until it ended or did you leave before it ended? 

□ Completed [Proceed to Question B-1] 

□ Left before it ended [Proceed to Question A-3] 

A-3. During what month did you stop attending the class or program? 

Month 



B.v pine?: Education and TfcAiNiNG 



B-1. Since the end of your class or program, have you enrolled in any other educational or training programs? 

□ Yes 

□ No [Proceed to Question C-1 ] 

B-2. Where are you enrolled? 

□ Other (Specify) 

B-3. In what type of class or classes are you now enrolled? [Do not read choices. Check all that apply.] 

□ English Language Skills 

□ GED/High School 

□ Vocational/Job Training 

□ Community College/College Level 

□ Citizenship 

□ Family literacy 

□ Other (Specify) 

□ DK/Refused 




National Reporting System: Conducting the Follow-up Survey 




C SECONDARY O^E£€WTIAL 




C-1 . Did you receive any diplomas, certificates, or degrees at the end of your class or since you left (TEACHER’S) 
class, such as the GED? 

□ Yes 

□ No [Proceed to Question D-1] 

□ DK/Refused [Proceed to Question D-1] 

C-2. What type of diploma/certificate/degree did you receive? [Do not read choices to respondent. Check ail that 
apply.] 

□ GED 

□ High School Diploma 

□ Certificate of Competence 

□ Associate’s Degree 

□ Bachelor’s Degree 

□ Other 

□ DK/Refused 



10.. Employment 



‘tV*VMiV>VV>V>ViViViVi'M'>^ViV/^VtV/tVtV^VnViVtVtV l ViViVi‘ t V t ViV l ViVtViiV>Vi‘iViVi ! 



D-1 . When you first enrolled in the class or program were you: [Read choices ] 

□ Employed at a paying job [Proceed to Question D-4] 

□ Not employed at a paying job and looking for a job [Proceed to Question D-2] 

□ Not employed and not looking for a job [Proceed to E-1] 

□ DK/RefusedfEnd/nferv/ewy 

D-2. While you were taking (TEACHER’S)’s class did you get a paying job? 

□ Yes 

If yes: What was the name of your employer? [Proceed to Question D-4] 

□ No 

D-3. Since you stopped taking the class, have you gotten a paying job? 

□ Yes 

If yes: What is the name of your employer? 

When did you first get a job after leaving the program? 

□ No [Proceed to Question E-1] 

D-4. Do you still have that job or do you now have a different job? 

□ Still have same job 

□ Have different job 

What is the name of your current employer? 

□ Lost job, unemployed 

□ DK/Refused 
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■l : E. Closing , 

you. v&y much lot taking the lime to; answer questions. Jour ansvtfersewil be -very' Ijeipfuj: {he . 
you.gave me w.fibe uss^lo help watt adult education programs better and mere useful to 
people like you who fraye attended onyouid like , to atteno such a program; 



E-1. Is there anything that I didn’t ask about that you’d like to say? 



BEST COPY AVAILABLE 
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Calling Log 



Interviewer: 



Date & Time 


Name 


Contact (who, nature of conversation, 
any messages left, etc.) 


Status 

(Interview completed, 
scheduled recall) 
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National Reporting System: Conducting the Follow-up Survey 



Sample Follow-up Survey for Core, Secondary, and Other Measures 



Hello. My name is. 



I work for _ 



. We're calling people who have recently attended 

our classes at our adult education programs to find out what happens to them after they leave us. We also want to 
know how you liked the classes you attended and how adult education classes have affected you, your family, and 
your job. 

It should take no longer than 15 minutes to answer my questions. Do you have time now for me to ask these 
questions? 'JReassure the respond ent that any information given to us will be strictly confidential). 









A-1. I understand that you were in (TEACHER’S NAME)’s class at (LOCATION). Is that correct? 

□ Yes 

□ No [ Obtain correct information] 



A-2. During what month and year did you enroll in this program? 
Month Year 

A-3. Did you attend the class/program until it ended? 

□ Yes [Proceed to question B-1] 

□ No [Proceed to question A-4] 

A-4. During what month did you stop attending the class or program? 
Month 



A-5. What was the main reason you stopped attending the class or program? [Do not read choices to 
respondent Check category that is most closely reiated to response.] 



□ Achieved reason for enrollment 

□ Completed class 

□ Illness/Incapacity 

□ Lack of child care 

□ Lack of transportation 

□ Family problems 

□ Time or location of services not feasible 

□ Lack of interest 

□ Instruction not helpful 

□ Instructor was not good 

□ Program didn’t satisfy personal goals 

□ Not satisfied with program 

□ Moved 

□ Entered employment 

□ Entered other education or training program 

□ Other (Specify: ) 

□ DK/Refused 
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B-1 . Did you receive any diplomas, certificates, or degrees since you took this class, such as a GED? 

□ Yes [Proceed to question B-2] 

□ No [Proceed to question C-1] 

□ DK/Refused [Proceed to question C-1] 

B-2. What type of diploma/certificate/degree did you receive? [Do not read choices to respondent Check 
all that apply] 

□ GED 

□ High School Diploma 

□ Certificate of Competence 

□ Associate’s Degree 

□ Bachelor’s Degree 

□ Other 

□ DK/Refused 



Other Education and Training 






C-1 . Since you stopped attending the class or program, have you enrolled in any other educational or 
training programs? 

□ Yes 

□ No [Proceed to question D-1] 

C-2. Where are you enrolled? 

□ Other (Specify) 

C-3. In what type of class or classes are you now enrolled? [Do not read choices. Check all that apply.] 

□ English Language Skills 

□ GED/High School 

□ Vocational/Job Training 

□ Community College/College Level 

□ Citizenship 

□ Family literacy 

□ Other (Specify) 

□ DK/Refused 




Employment 




D-1 . While you were enrolled in the class or program, were you receiving any type of public assistance, 
such as food stamps or welfare benefits? 

□ Yes 

□ No [Proceed to question D-3] 

□ DK/Refused [Proceed to question D-3] 
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D-2. Are you currently receiving this type of public assistance? 

□ Yes 

□ No 

□ DK/Refused 

D-3. When you first enrolled in the class or program, were you: [Read choices.] 

□ Employed at a paying job [Proceed to question D-6] 

□ Not employed at a paying job and looking for a job [Proceed to question D-4] 

□ Not employed and not looking for a job [Proceed to question E-1 ] 

□ DK/Refused [ Proceed to question E-1] 

D-4. While you were taking this class, did you get a paying job? 

□ Yes 

If yes: What was the name of your employer? [Proceed to 

Question D-6] 

□ No [Proceed to question D-5] 

D-5. Since you stopped taking this class, have you gotten a paying job? 

□ Yes 

If yes: What is the name of your employer? [Proceed to 

Question D-6] 



When did you first get a job after leaving the program? 

□ No [Proceed to E-1] 

D-6. Do you still have the same job, have a different job, or have no current job? 

□ Still have the same job 

□ Have a different job 

What is the name of your current employer? 

□ Have no job, unemployed 

□ DK/Refused 






E-1. Compared to before you attend the class, have you increased your attendance or activities in any of 
the following: [Read choices. Check all that apply.] 

□ Neighborhood meetings 

□ Meetings of political groups 

□ Volunteer work or meetings for community organizations 

(List: ) 

□ Do not got to meetings or volunteer 

□ DK/Refused 

E-2. Did you register to vote or vote for the first time since you attended the class? 

□ Yes 

□ No 

□ DK/Refused 
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F-1 . Do you live with children who are 12 years old or younger? 

□ Yes 

□ No [Proceed to question G-1] 

F-2. Since you attended the class, how much do you read with your children compared to before you 
attended the class? Do you: 

□ Read with children about the same as before 

□ Read with children more than before 

□ Read with children less than before 

□ Do not read with children at all 

□ DK/Refused 

F-3. How often do you visit the library with your child/children now compared to before attending the 
program? Do you: 

□ Go more often 

□ Go the same amount 

□ Go less often 

□ Not go at all 

□ DK/Refusal 

F-4. Is/are the child/children in your home attending school? 

□ Yes [Proceed to question F-5] 

□ No [Proceed to question G- 1 ] 

F-5. Compared to before you attended the class, how much time do you spend helping the school-aged 
children in your home with homework? Do you: 

□ Help about the same 

□ Help more than before 

□ Help less than before 

□ Not help at all 

□ DK/Refused 

F-6. Compared to before you attended the class, how many of your children’s school activities including 
parent/teacher conferences and school assemblies have you gone to? 

□ Attend about the same 

□ Attend more activities 

□ Attend few activities 

□ Do not attend activities 

□ DK/Refused 



| 5gi7S' T A CTTc?a J \XW i H 



G-1. What is your general opinion of the quality of the class you attended? Is it unacceptable, not very 
good, satisfactory, or excellent? 

□ Unacceptable 

□ Not very good 

□ Good 

□ Excellent 

□ DK/Refused 
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G-2. Did (TEACHER’S NAME)’s class meet the expectations you had for it before you enrolled in it? 

□ Yes 

□ No 

□ DK/Refused 

G-3. Are you not at all likely, somewhat likely, or extremely likely to attend another class or program 
offered by (PROGRAM/CLASS ORGANIZER )1 

□ Not at all likely 

□ Somewhat likely 

□ Extremely likely 

□ DK/Refused 

G-4. What did you like about this class or program? [List all responses.] 



G-5. What did you not like about this class or program? [List all responses.] 



' CLOSING .. • : 

ij^iank you vary much for faking ins tiffie to answer rrty questions. Your answers are very helpful. Tine 
i information you gave me wilt bs used to help make adult education orograms better and more useful to 
i pe^QjifeyQtkh^BE'e ettshssc program. 



H-1. 



Is there anything that I didn't ask about that you'd like to say? ' 



Model Procedures for Conducting the 
Local Follow-up Survey 



This section describes model procedures for conducting a telephone survey designed 
to collect the NRS follow-up measures. The model is offered as guidance to states in 
designing and conducting the follow-up survey. These procedures are not required \ 
however , and states may develop their own procedures for conducting the survey , as long as 
they meet the NRS requirements described in this document. 

The crucial activities to a conducting a telephone survey that produces valid data are 
to: 



1. Draw a sample of students that reflects the students who attend your program 
with one or more of the four core outcomes as main or secondary goals; 

2. Reach die students sampled and obtain the information from a large majority of 
them so as not to invalidate the sample; and 

3. Train telephone interviewers so that all interviewers ask the survey questions 
correctly and reliably. 

Selecting the Sample 

The procedures below present a method for randomly selecting a sample of 300 
students who left the program. 

1. Generate a list from your database of names, with telephone numbers and contact 
information, of students who have left the program, organized by the four core 
follow-up measures. Use students’ stated main or secondary goals to identfy the 
groups. You should also have the exit quarter for students with employment 
goals. You may use separate lists for each of the four follow-up measures or a 
single list with all students. 

2. Go through the list to identify any individuals who do not have a telephone 
number or any contact information. Cross these names off your list. 

3. We want to compute a sampling fraction that will give us a 300 student sample. 
The sampling fraction is computed by dividing the total number of students to be 
sampled by the desired sample size. If we have 600 students, the sampling 
fraction is 600/300 = 2. If the sampling fraction is not a whole number, it should 
be rounded to the nearest whole number. 

4. From your list, count down the number of students determined by the sampling 
fraction and include that student in the sample and continue this way throughout 
the entire list. For example, in this example the sampling fraction is 2, so include 
the second student on die list and every other student thereafter. When you are 
finished, you will have a sample of 300 students. This is your primary sample. 
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5. Create a backup sample of 50 percent more students than your primary sample. 
Randomly select the backup sample in the same way as the original sample. 
Compute die sampling fraction by dividing the number of students you need by 
the number of students remaining on die list. Use this number to select every n lh 
student from the list. Make sure you have a backup sample sufficient for each of 
the four core outcome measures. The backup sample is used to replace students 
from the primary sample who cannot be reached after four attempts (see below). 
If there are fewer students remaining on your list than you need for the backup 
sample after you have selected your primary sample, all of these students will be 
included in the backup sample. 

Survey Procedures 

Once you have your sample, you can begin calling students and administering the 
survey. Call each person on your primary sample list. If you cannot reach a person despite 
your best efforts, replace that student with a student from the backup sample. 

As you conduct the survey, it is very important to the integrity of the data collected to 
know how many people in the sample were not reached, how many refused to participate, 
and what the reasons for refusal were. For this reason, maintain a calling log during the 
conduct of the survey. Entries in die log should contain the date and time of each call, the 
name of the caller, and information about the call, including: the name of the respondent, 
whether die person was reached, messages left, whether the interview occurred, and 
explanations for why it did not. The logs should be checked daily to identify respondents 
who need to be re-called. They should also be checked against the list of learners in die 
sample to make sure all members of the sample are being called and contacted. Callers 
should promptly make a log entry for every telephone call they make, whether or not die 
adult learner was reached. This Appendix includes a sample calling log. 

The validity of the survey depends on reaching all or at least a majority of the 
students in the sample. There will be many difficulties, however, in reaching all of the 
students in the sample. The following section describes some of the most common 
difficulties in reaching people for a telephone survey and guidance on how to resolve diese 
problems. 



Problems Reaching Learners on the Telephone 

In most data collection activities, diere are predictable kinds of problems that may be 
encountered. Interviewers may be unable to reach the correct person, the learner may not 
want to speak to the caller, or they may have a protective family. Additionally, learners may 
not want to answer some or all survey items; they may be hostile, confused, or just harried. 
Further, callers may be required to answer questions that they are not equipped to answer. 

Interviewers should have a resource person available who can assist with difficult 
interviews or respondents, and complicated questions. This person should have thorough 
familiarity with the NRS and the procedures used to conduct telephone interviews. She or he 
should monitor interviewer telephone logs, provide general oversight during the interviewing 
process, and could also be responsible for the training. 
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Accommodation for other languages. Since the sample may include ESL students 
and other non-native English speakers, interviewers are likely to encounter a language barrier 
in the course of data collection. Every effort must be taken to collect information from all 
non-English speakers included in the sample. Accomplishing this may require the program 
to translate the survey and use interviewers who are fluent in the languages that may be 
encountered during the interviews. The NRS has Spanish and Vietnamese versions of the 
model survey, available on request. 

When the student cannot be reached immediately. A gatekeeper is a person or 
situation that stands between you and the person with whom you need to talk. Common 
gatekeepers are family members, and even answering machines. 

• Reaching a family member or other person 

3. Leave a message. The message should be as follows: 

• Interviewer’s name 

• Calling from (name of program) 

• Calling in reference to the adult education program the person attended 

• Interviewer will call back at another time. 

4. Ask a few questions: 

• When is the learner expected back? 

• What and when is the best way to reach her/him? 

5. Wait for no more than two days between callbacks. 

6. If multiple messages (more than 3-4) have been left, but the learner has not been 
contacted, the learner should be officially listed as a non-respondent on the calling log 
sheet and replaced from the backup sample. 

• Reaching voice mail or an answering machine 

7. Leave a message. The message should be as follows: 

• Interviewer name and where interviewer is calling from (name of 
program) 

• Calling in reference to the adult education program the person attended 

• Interviewer will call back at another time 

8. Wait for no more than two days between callbacks. 
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9. If multiple messages (more than 3-4) have been left, but the learner has not been 
contacted, the learner should be officially listed as a non-respondent on the calling log 
sheet and replaced from the backup sample. 

• Reaching a non-working number or a number that just rings 

10. Non-working number should be noted on the calling log sheet as not working. 

11. If the number just rings, the day and time the interviewer called should be noted on the 
log sheet, and the learner should be called at a different time. If multiple calls (more than 
3-4) are made at different times of the day, and there is still no answer, the learner should 
be officially listed as a non-respondent on the calling log sheet and replaced from the 
backup sample. 

Dealing with refusals. The goal of telephone interviews is to obtain information 
from all the people contacted. However, some interviewees may be initially reluctant to 
participate in the survey. The interviewer should try to “convert” refusals whenever possible; 
callers should, however, never become belligerent or upset or insist that a person complete 
the survey. 

The best way to handle a refusal is for the caller to present himself or herself as 
confident and proud of the work they are doing. The interviewer should indicate that this 
survey is an important way of providing information to the State Department of Education 
and the adult education program, and decisions about adult education will be made based on 
this information. 

There are several points in the interview when callers may encounter refusals or 
reluctance. The following examples provide ways to handle this. 

Initial refusal. When learners are first reached, they may not be prepared to speak 
with the interviewer. They may be very busy. If this is the case: 

• Ask about the timing: I’m sorry we reached you at a bad time. When might be a 
more convenient time to reach you? Possible solutions include offering to call 
them a week later, a month later, etc., as long as this is recorded so that the 
follow-up call is made. 

• When the learner has been reached, but absolutely refuses to participate, a 
complete description should be recorded on the calling log and given to the 
resource person for further attempts. 

Confusion-based refusal. Adult learners who are contacted may be confused or 
wary about how the information collected in the interview will be used. For this reason, they 
may refuse to take part in the interview. 

• If the learner wants to know why the survey is being conducted, the interviewer 
should explain the purpose of the study, emphasizing that the information 
collected has important implications for the national adult education program, as 
well as for the program she or he attended. 
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• If the learner wants to know how their information will be used, the interviewer 
should assure die learner diat the data will be compiled to find out how well adult 
education programs are performing throughout the country and to improve 
program services. Further, all of the answers that the learner gives will be kept 
confidential and that no names or other identifying information will be associated 
with their answers. Learners should also be assured that they were chosen 
randomly from the pool of adult learners in the state. 

Time or burden-based refusal. This type of refusal can occur early in the 
interview, or at a later point. Interviewees may be pressed for time and may try to terminate 
the interview. If this is the case: 

• The interviewer should point out that the survey will only take 10-15 minutes, 
acknowledge that the learner's time is really important, and tell them that their 
responses to the survey questions would be really helpful: I understand that your 
time is important. We really appreciate your input on this issue. It is important 
to get die perspective of adult education students. 

• The interviewer should tell diem about the sampling process: Of the [number] 
students diat attended the adult education program, you have been selected as one 
of only [number] to represent die program. Your help is important to us. 

If the respondent is still reluctant, one other strategy may be helpful: 

• The interviewer should try to arrange an alternate time: Might there be a better or 
more convenient time to contact you? 

If none of these strategies is successful, the interviewer should NOT try to persuade 
the learner further. The learner should be thanked for their patience, and told that die caller 
appreciates all the demands on their time. The interviewer should then record a complete 
description on die calling-log contact sheet and the student should be replaced from the 
backup sample. 

Training 

Staff members who will be conducting the telephone interviews should be trained to 
ensure the integrity of the data collected. To collect valid and reliable data, interviewers 
must be dioroughly familiar with both the process of interviewing and the materials to be 
used for collecting data. The actual training, therefore, can be characterized as having two 
components: the process of conducting telephone interviews and the purpose and structure of 
the NRS. This section provides suggestions on appropriate training activities. 

Focus of Training 

Regardless of the survey, any errors, biases, or inconsistencies on the part of the 
interviewer result in some degree of survey error. It should be a goal to minimize this error. 
Trained interviewers are much more likely to accomplish diis goal. The desired result is high 
quality data, so that data are comparable from one interview to another, as well as from one 
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state to another. The following guidelines should help minimize survey error, and should 
thus be conveyed to the interviewers during their training. 

1. The interviewing process should be standardized. To ensure that this occurs, 
interviewers must read the questions exactly as written and follow the 
instructions on the survey instrument. 

2. Interviewers should avoid biasing answers by not showing criticism, surprise, 
approval, disapproval, and/or annoyance at a response; recording answers 
promptly and accurately; and probing for clarification when necessary. 

3. Interviews should be completed in the time promised to the respondents. The 
interview is designed to take about 10 minutes. 

4. Interviewers must be familiar with the material, including the meaning of 
individual questions and die definitions of words and phrases contained in the 
survey instrument. 

5. Administrative issues should be attended to as soon as possible, including 
making a record of EVERY call made, even if the interviewers reached a 
wrong number, if nobody answered, or if a message was left. 

6. Interviewers should have a thorough understanding of the purpose and 
structure of the NRS and the pilot, as explained in this manual. 

Conducting the Training 

Training interviewers can take many forms, including workshops and meetings. 

There are, however, a few techniques which will make die training more meaningful, and 
thus make the data collected more useful and comparable between states. Among these 
techniques are: 

• Going over the protocol question-by-question. This will give interviewers a 
familiarity with the questions and answers they are likely to get during their 
telephone calls. It will also allow diem to become comfortable with die decisions that 
must be made as die interview begins. 

• Conducting mock interviews with adult education office staff or teachers. This 
simulates real world conditions, giving interviewers valuable practice on how to 
conduct interviews. It also allows adult education office staff to identify issues that 
were not made clear earlier in the training process and to identify problems with die 
data collection procedures in place. 

• Conducting a mini-pilot test with students not included in the official NRS 
sample in the state. This activity will identify previously unconsidered issues and 
provide the most realistic training for the interviewers. It is an excellent last step 
prior to officially collecting data. 
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Computing the State Rate from Local 

Samples 



Appendix C 

Computing the State Average and Confidence Interval from Local Data 

States must report the overall state average for each of the four follow-up 
outcome measures to report to the U.S. Department of Education. To calculate this 
average, states must combine the averages from all of the separate local programs into a 
single average. If any local programs used a sample of students for any of the outcome 
measures, as will probably be the case for most or all states, the state must also report the 
overall confidence interval for the state average. The state must also report the state 
response rate. This Appendix explains the procedure for calculating these numbers. 

Computing the State Average 

Each local program must report the following information to the state: 

L N — The total number of students in the outcome group (i.e., total number of students 
who had a main or secondary goal related to the outcome). 

2. n — The total number of students who responded to the survey. 

3. r — The percentage of students who obtained the outcome. 

4. The size of the sample drawn, if the program sampled. For programs that did not 
sample, this number will be the same as N (number 1, above). The response rate is 
the number who responded (n) over the total number (N). 

To determine the state average, average each program measure according to the 
formula: 



S 
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This formula simply means that the percentage (r) from each program is 
multiplied by the total number of students in the outcome group (N) from each program 
and divided by the total number (N) and then summed for each program. For example, 
suppose one program has 150 students with a goal of obtain employment, tries to contact 
them all (does not sample), reaches 100, and finds that 25 percent of the 100 are 
employed. A second program has 2,000 students with a goal of obtain employment, 
samples 600 and reaches 300. Of these 300, 30 percent are employed. These programs 
then report to the state the following: 
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Program 1 


Program 2 


N = 150 


N = 2,000 


n = 100 


n = 300 


r = 25 


r= 30 


The state average (R) is then: 




R = (25 * 150) + (30 * 2000) = 29.65 





150 + 2000 



Notice that this state average, 29.65 percent, is much closer to the average of 
Program 2, since Program 2 has more students and thus represents more of the state adult 
education student population who have an employment goal. 

The state would also report the response rate for both programs. For Program 1 
the response rate is 100/150 = 67 percent and for Program 2 it is 300/600 = 50 percent. 
The combined response rate is 100 + 300 = 53.3 

150 + 600 

Computing Confidence Intervals 

If any program samples students, as is the case with Program 2 in the example 
above, the state must also compute and report the overall confidence interval for the state 
average. The confidence interval represents the amount of uncertainty to the state 
estimate that results from the sampling error. 

To compute the state confidence interval, the state must first compute the standard 
error (SE) for each program that sampled. SE is computed according to the formula: 



/ r (100 — r) 
y n 

The confidence interval is computed by doubling the standard error and then adding it to 
the average to get the upper bound (rhi) and subtracting it from the average to get the 
lower bound (ri ow ). 

If the program does not sample, there is no standard error due to sampling and ri 0 
is the same as i),i, which is the same as the state average. In the example given above. 
Program 1 did not sample, so its SE = 0. Consequently: 



ERiC 



51 



Program 1 



Ni = 150 

m = 150 

r, = 25% 

ri, io = 25% 

ri, hi = 25% 



For Program 2, which did sample, the state would first have to compute the standard 

error: 

Vr2(100-n)/«2=V30(100-30)/300.=2.65 

Thus for Program 2: 

N 2 = 2000 

n 2 = 300 

r 2 = 30% 

r 2 ,io = 30 -2*2.65 = 24.7% 

r 2 ,hi = 30+2*2.65 = 35.3% 

In this example, the lower and upper bounds of the confidence interval on the overall 

state rate are thus: 

Rstatejio = 25* 150 + 24.7*2000 = 24.72 

150 + 2000 

R,tat*hi= 25* 150 + 35.3*2000 = 34.58 
150 + 2000 



In this example, the state would then report that the number of students who 
entered with a goal of obtain , employment who obtained employment ranges from about 
24.72 to about 34.58 percent. Another way to say this is that the average proportion of 
students who obtain employment is 29.65 percent, plus or minus 4.93. 
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